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EXECUTIVE  SUMMARY 


This  is  a  final  report  of  a  three-phase  project.  In  this  report,  a  series  of  recommendations  are 
outlined  that  address  a  number  of  shortcomings  in  how  aircraft  manufacturers  develop  aircraft 
maintenance  documentation.  In  Phase  1 ,  a  number  of  problems  were  documented,  including  the 
reactive  rather  than  proactive  use  of  user  evaluations,  the  limited  use  of  user  input  and  procedure 
validation,  the  absence  of  systematic  attempts  to  track  errors,  and  the  lack  of  standards  for 
measuring  document  quality.  In  Phase  2,  a  survey  of  aircraft  technicians  revealed  that,  although 
user  evaluations  of  the  accuracy  and  quality  of  technical  manuals  are  generally  good,  the  manuals 
themselves  were  noted  as  having  poor  usability.  In  light  of  these  deficiencies,  it  is  recommended 
that  (1)  manufacturers  and  operators  work  to  facilitate  communication  between  the  technicians 
submitting  change  requests  and  the  technical  writers  making  the  changes  to  ensure  prompt 
feedback  regarding  actions  to  be  taken,  (2)  maintenance  procedures  be  validated  by  technicians 
using  standard  human  factors  techniques,  (3)  the  industry  cooperate  in  the  development  of  a 
system  akin  to  the  Maintenance  Steering  Group  (MSG)  to  identify  maintenance  procedures  that 
should  be  systematically  validated  and,  (4)  manufacturers  should  maintain  databases  with  a 
history  of  user-reported  errors,  feedback  to  the  user,  and  actions  taken.  By  tracking  the  history  of 
user  error  reports,  manufacturers  can  target  maintenance  procedures  for  validation  that  have  the 
greatest  potential  impact  on  safety  and/or  economics.  This  information  can  also  be  used  to 
develop  more  meaningful  metrics  of  manual  quality.  Finally,  an  example  is  described  (using  the 
MSG-3  process)  of  how  these  recommendations  may  be  implemented. 
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1.  INTRODUCTION. 


The  Federal  Aviation  Administration  (FAA)  has  sponsored  a  number  of  research  projects  [1  and 
2]  addressing  issues  related  to  maintenance  data  and  ways  to  improve  the  reliability  of  this 
documentation.  FAA  research  is  motivated  by  several  factors  including  the  FAA  goal  [3]  to 
“reduce  the  fatal  accident  rate  for  commercial  air  carriers  by  9  percent  from  a  1994-1996  baseline 
of  0.037  fatal  accidents  per  100,000  flight  hours”  and  accident  investigations  wherein 
maintenance  procedures  and  documentation  were  cited  as  potential  contributing  factors.  Prior  to 
the  events  of  September  11,  2001,  it  was  projected  that  by  2010,  given  the  forecasted  increase  in 
air  traffic  and  the  current  accident  rate,  one  aircraft  accident  would  occur  each  day.  The  current 
project  is  a  small  part  of  a  larger  multifaceted  FAA  research  program  investigating  ways  to 
improve  aircraft  safety. 

The  current  report  summarizes  a  set  of  recommendations  derived  from  an  FAA-sponsored 
research  project  [4  and  5].  The  purpose  of  the  research  project  was  (1)  to  investigate  how 
maintenance  documentation  is  developed  by  original  equipment  manufacturers  (OEMs)  and 
(2)  to  investigate  user  satisfaction  and  usability  problems  users  encounter  when  using  the 
documentation.  Technical  reports  describing  the  Phase  1  and  2  findings  of  this  research  project 
[4  and  5]  are  available  from  the  FAA. 

In  brief,  the  earlier  investigations  revealed  that  each  manufacturer  uses  a  set  of  unique 
procedures,  computer  applications,  and  development  guidelines  for  their  technical  manuals. 
They  each  maintain  detailed  procedures  and  sophisticated  software  applications  to  minimize  the 
inclusion  of  incorrect  technical  information  in  fielded  manuals.  Once  published,  each 
manufacturer  has  their  own  established  procedures  for  correcting  errors  in  the  published  manuals. 
These  procedures  include  methods  for  prioritizing  user-reported  errors,  which  determine  the 
order  in  which  the  errors  will  be  corrected.  Errors  judged  to  affect  safety  of  flight  are  addressed 
immediately,  whereas  other  errors  receiving  lower  priority  ratings  may  or  may  not  be  addressed  if 
more  significant  issues  are  subsequently  reported.  Although  unique,  no  one  manufacturer’s  set 
of  procedures  or  guidelines  was  identified  by  users  as  resulting  in  a  decidedly  better  manual. 

The  manufacturers  maintain  databases,  cataloging  user-reported  errors.  Flowever,  these 
databases  are  rarely  used  in  any  systematic  way  to  identify  trends  in  error  reports  or  what  caused 
the  errors.  Rather,  this  information  is  used  to  keep  track  of  which  errors  need  to  be  corrected  and 
as  a  simple  way  to  measure  manual  quality  (i.e.,  year-to-year  comparisons  of  the  number  of 
errors).  The  manufacturers  perform  limited  testing  of  documentation  to  ensure  that  maintenance 
procedures  are  satisfactory  from  the  user’s  perspective.  The  typical  document  testing  or 
evaluations  that  are  performed  are  often  less  than  satisfactory,  which  is  based  on  the  judgement 
of  the  principal  investigator  at  WSU,  for  a  number  of  reasons.  For  instance,  evaluations  are  not 
systematic,  do  not  employ  standard  usability  techniques,  and  are  often  performed  using  company 
employees  or  very  experienced  technicians  which  may  limit  the  applicability  of  the  results  to 
other  populations  of  users.  The  choice  of  participants  for  the  usability  evaluations  is  important  to 
ensure  the  results  will  be  widely  applicable.  Mechanics  with  different  levels  of  experience  may 
identify  different  problems  with  procedures.  Experienced  users  can  rely  on  their  knowledge  of  a 
system  to  understand  the  intent  of  a  writer,  whereas  less  experienced  mechanics  may  rely  to  a 
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greater  degree  on  the  specific  step-by-step  directions  outlined  by  the  writer.  The  input  of 
experienced  technicians  may  be  valuable  during  the  initial  development  of  procedures  because 
they  identity  critical  information  and  provide  insight  into  the  most  efficient  way  to  perform  a 
procedure.  However,  the  participation  of  less  experienced  mechanics  is  necessary  in  later  stages 
of  development  to  ensure  the  results  of  the  evaluation  will  apply  to  the  broader  population  of 
technicians. 

An  earlier  survey  [5]  of  user  experiences  indicated  that  the  primary  problem  with  manuals  is  not 
the  quality  of  the  technical  information,  or  the  number  of  errors  in  the  manuals,  but  rather  the 
usability  (i.e.,  ease  of  use)  of  the  manual  and  the  procedures  described  within.  More  specifically, 
users  cited  numerous  examples  where  maintenance  procedures  were  inefficient  and/or  failed  to 
consider  the  demands  of  the  maintenance  environment.  For  example,  when  working  on  an 
electrically  powered  system  the  technician  might  be  instructed  at  different  times  to  remove  circuit 
breakers  located  in  the  cockpit  rather  than  instructing  them  to  remove  them  all  at  one  time. 
Technicians  also  gave  examples  of  how  instructions  could  be  reorganized  to  improve  workflow 
and  reduce  the  time  to  perform  a  procedure.  The  results  of  Phases  1  and  2  [4  and  5]  illustrate  the 
need  to  (1)  improve  communication  between  writers  and  users  (i.e.,  technicians),  (2)  consider  the 
requirements  of  the  user,  and  (3)  perform  more  detailed  tracking  and  analysis  during  the  manual 
development  and  revision  process. 

Many  of  the  difficulties  that  users  have  with  maintenance  documentation  arise  because  the 
documentation  reflects  the  priorities  of  the  manufacturers.  The  manufacturers  are  primarily 
concerned  with  the  production  and  delivery  of  aircraft.  Matters  regarding  documentation  revolve 
around  the  timely  development  of  documentation  to  ensure  delivery  schedules  are  not  impacted. 
This  has  resulted  in  a  system  that  is  reactive  rather  than  proactive  in  nature.  This  is  partly  due  to 
the  lower  priority  given  to  technical  documentation  development  by  most  manufacturers  as 
reflected  in  their  development  schedules,  budgets,  and  the  limited  access  technical  writers  have 
to  the  aircraft.  With  new  aircraft,  pressures  of  meeting  production  schedules  force  an  uneasy 
balance  between  two  competing  demands — manual  quality  and  timely  delivery.  Manufacturers 
acknowledge  that  some  problems  may  exist  in  newly  fielded  manuals  but  that  most  errors  will  be 
identified  quickly  and  addressed  in  future  revisions.  Through  this  process,  it  is  assumed  that  the 
manual  will  be  improved  as  errors  or  problems  are  identified  and  corrected.  Simply  correcting 
errors  may  not  improve  the  manuals.  When  manufacturers  analyze  errors  reported  in  fielded 
manuals  they  tend  to  focus  on  where  the  errors  originated  in  the  development  process. 
Consequently,  there  is  little  understanding  of  the  needs  of  the  technicians  who  must  use  the 
documentation,  their  problems  with  the  documentation,  or  ways  of  eliminating  those  problems. 
For  this  reason,  usability  problems  are  likely  to  recur  in  new  and  revised  manuals.  Without 
performing  an  analysis  of  the  error  reports  from  the  field,  there  is  no  way  of  determining  how 
changes  in  the  writing  process  would  affect  development  costs.  A  series  of  recommendations 
addressing  these  and  other  issues  are  described  in  the  remainder  of  this  document.  In  addition,  a 
system  for  implementing  these  recommendations  is  also  described. 
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2.  RECOMMENDATIONS. 


2.1  IMPROVED  COMMUNICATION  BETWEEN  USERS  AND  TECHNICAL  WRITERS. 

The  purpose  of  technical  documentation  is  to  support  the  maintenance  of  aircraft  by  providing 
relevant  information  to  technicians  who  perform  maintenance  tasks.  As  noted  above,  the  current 
system  is  driven  by  the  need  to  meet  the  regulations,  requiring  only  that  information  be  provided 
for  all  procedures  necessary  to  maintain  continued  airworthiness.  The  emphasis  is  on  providing 
the  information  and  not  the  needs  of  the  user.  The  fact  is,  the  goal  of  technical  documentation  is 
to  support  the  needs  of  the  technicians  performing  maintenance  on  the  aircraft.  Yet,  most 
manufacturers  assume  that  users  will  report  quality  control  problems  to  them  when  they  are 
identified.  This  is  true  despite  the  fact  that  the  system  does  little  to  encourage  this  behavior. 
Change  requests  made  by  technicians  are  typically  processed  by  several  intermediaries,  including 
company  representatives,  engineers,  and  managers  from  the  operator  and  the  manufacturer, 
before  reaching  a  technical  writer.  The  same  occurs  when  a  technician  receives  a  reply  from  a 
technical  writer.  In  personal  interviews,  technicians  reported  frustration  with  the  current  system, 
citing  lack  of  feedback  after  submitting  change  requests  and  noting  that  the  same  errors 
continued  to  appear  in  later  manual  revisions.  The  delayed  or  absence  of  feedback  does  not 
encourage  users  to  report  problems.  Manufacturers  and  operators  should  work  together  to 
facilitate  the  submission  of  change  requests  and  ensure  that  users  receive  prompt  responses  to 
their  requests  regardless  of  whether  or  not  the  procedure  will  be  changed. 

2.2  VALIDATION  OF  MAINTENANCE  PROCEDURES. 

As  part  of  an  FAA-sponsored  research  program  [3],  a  formatting  tool  called  the  Document 
Design  Aid  (DDA)  was  developed  to  improve  the  presentation  of  maintenance  information  by 
using  workcards.  Workcards  are  documents  that  are  used  by  a  technician  or  inspector  to  specify 
the  individual  steps  required  to  perform  an  inspection  or  repair  procedure.  This  tool  serves  as  a 
guideline  for  the  development  of  workcards  that  take  into  account  how  humans  read,  process, 
and  interpret  written  information  in  order  to  improve  their  usability.  For  instance,  the  DDA 
provides  design  guidelines  on  information  readability,  context,  organization,  and  consideration  of 
factors  affecting  the  handling  (e.g.,  weight,  size,  etc.,  of  a  manual  or  card)  of  the  documents. 
Although  the  DDA  can  improve  the  usability  of  maintenance  documentation,  it  does  not  address 
the  content  of  the  procedure  or  specify  how  to  evaluate  the  documentation.  In  light  of  earlier 
findings,  the  lack  of  a  systematic  evaluation  of  the  reliability  of  maintenance  documentation  is  a 
critical  deficiency  in  the  existing  manual  development  process. 

A  necessary  complement  to  design  guidelines  is  techniques  to  evaluate  the  reliability  of 
maintenance  documentation  as  it  is  developed.  The  fact  that  users  rate  the  problem  of  incorrect 
data  in  the  manuals  as  being  low  is  not  a  surprise  given  the  substantial  efforts  [4]  made  to 
minimize  their  inclusion  in  the  manual.  In  order  for  a  technical  error  to  contribute  to  a 
maintenance  mishap,  the  error  must  occur  in  documentation  pertaining  to  safety-critical  systems 
or  processes,  and  the  discrepancy  between  the  correct  and  incorrect  information  must  be 
sufficiently  large  enough  to  cause  a  problem,  but  not  so  drastic  that  it  is  immediately  identifiable 
as  incorrect.  What  is  more  likely  are  cases  of  missing  or  incomplete  data  and  procedures  that  are 
out  of  order,  unclear,  difficult  to  interpret,  or  awkward  to  perform.  In  these  cases,  the  technician 
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must  rely  on  their  own  judgment  and  experience  to  determine  the  best  way  to  proceed.  When 
interviewing  technicians,  the  most  common  complaint  they  had  was  frequently  performing 
procedural  steps  out  of  order  because  the  steps  could  not  be  performed  as  written.  The  procedure 
may  include  all  of  the  necessary  information  and  may  be  factually  accurate;  however,  it  may  be 
difficult  for  the  technician  to  perform  the  procedure  correctly  because  of  the  unclear  instructions. 
In  many  cases,  these  problems  could  be  identified  during  the  development  process  using  a  variety 
of  techniques  ranging  from  verification  by  an  expert  to  validation  of  the  procedure. 

The  term  validation  refers  to  a  process  where  a  technician  performs  a  procedure  as  described  in 
the  maintenance  manual  under  standard  conditions  (i.e.,  at  a  maintenance  base  or  at  the  gate 
terminal  using  typical  tools).  The  purpose  of  the  validation  is  to  determine  whether  procedures 
can  be  performed  in  a  safe  and  efficient  manner  as  written.  Although  some  manufacturers  have 
validated  procedures  in  the  past,  this  was  normally  not  performed  in  a  methodical  or  controlled 
manner.  Typically,  a  technical  writer  might  observe  a  technician  on  the  assembly  line  or  at  a 
company  service  center  performing  the  procedure  of  interest.  The  results  from  this  validated 
procedure  may  not  represent  real-world  conditions  for  a  number  of  reasons.  First,  a  company 
employee  may  be  very  familiar  with  the  design  and  operation  of  a  specific  component  and  may 
have  experience  performing  this  procedure  on  similar  aircraft.  Consequently,  they  may  report 
fewer  problems  because  they  have  a  level  of  experience  and  familiarity  that  other  users  may  not 
have.  Second,  the  validation  may  be  performed  on  the  assembly  line  on  a  new  aircraft  using 
special  tools  available  only  at  the  manufacturing  site.  A  technician  in  the  field  is  faced  with  the 
dirt  and  grime  typical  of  in-service  aircraft  or  poor  environmental  conditions  (low  light  or  poor 
weather)  while  using  a  different  or  limited  set  of  tools.  Under  these  conditions,  it  is  not 
surprising  that  they  report  more  difficulty  performing  the  same  procedures. 

Not  all  maintenance  procedures  may  require  formal  validation.  A  number  of  other  evaluation 
techniques  may  be  used,  ranging  from  simple  proofreading  by  a  colleague,  to  a  technician 
reviewing  the  procedural  steps,  or  simulated  performance  by  a  technician.  Procedures  that  are 
modifications  of  existing  procedures,  not  safety-related  or  rarely  performed  (e.g.,  procedures  for 
rodent  removal  or  volcanic  ash  encounters)  may  be  subject  to  proofreading  only.  Other 
procedures  that  may  be  more  involved  or  require  special  equipment  and/or  parts  may  be 
evaluated  by  having  maintenance  personnel  verify  the  instructions  without  physically  removing 
components.  Procedures  can  be  evaluated  in  this  way  to  ensure  there  are  no  physical 
obstructions  that  prevent  the  procedure  to  be  performed  as  described,  that  the  correct  tools  and 
part  numbers  are  identified,  and  that  the  instructions  are  complete.  Formal  validation  would  be 
reserved  for  procedures  that  are  performed  frequently,  prone  to  mistakes,  or  are  deemed  safety 
critical. 

2.3  DATABASE.  TRACKING.  AND  ANALYSIS  OF  MANUAL  RELIABILITY. 

An  essential  aspect  of  determining  the  reliability  of  a  system  is  the  collection  and  analysis  of 
operating  data.  To  track  document  reliability,  it  is  necessary  to  develop  and  maintain  a  database 
consisting  of  a  history  of  user-reported  errors,  feedback  to  their  reports,  and  any  follow-up 
information.  The  majority  of  manufacturers  use  feedback  from  operators  to  identify  potential 
problems  in  their  manuals.  If  it  is  determined  that  a  revision  of  a  procedure  is  necessary,  then  a 
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change  is  scheduled  and  the  error  report  is  eventually  deleted.  This  purging  of  feedback  data  is 
not  necessarily  an  attempt  to  hide  problems,  but  rather  indicative  of  a  reactive  approach  toward 
fixing  errors.  Documents  are  released,  problems  are  identified,  revisions  are  made  to  the 
document  based  on  those  problems,  and  then  re-released.  The  assumption  of  this  type  of  reactive 
approach  is  that  technical  documents  will  improve  with  age.  However,  because  of  the  limited 
depth  of  data,  it  is  not  possible  to  track  trends  over  time.  This  approach  can  be  of  considerable 
cost  to  the  manufacturer,  operator,  and  industry  as  a  whole  because  similar  problems  may 
continue  to  appear.  By  tracking  the  number  and  type  of  user  responses,  trends  can  be  identified 
that  can  guide  efforts  to  improve  manuals  and  serve  as  useful  metrics  of  manual  quality. 
Manufacturers  could  save  resources  by  identifying  and  validating  procedures  that  have 
historically  been  problematic.  Operators  could  also  benefit  by  identifying  the  personnel, 
equipment,  and  environmental  variables  that  contribute  to  usability  problems  and  respond 
accordingly. 

3.  IMPLEMENTATION  OF  RECOMMENDATIONS. 

3.1  DEMONSTRATING  THE  BENEFITS  OF  VALIDATION. 

The  potential  impact  of  poor  maintenance  documentation  on  safety  is  of  primary  concern  to 
operators  and  manufacturers.  Less  widely  acknowledged  is  the  cost  burden  of  poor  maintenance 
documentation  on  the  aircraft  industry.  From  the  perspective  of  the  manufacturer,  poor 
documentation  increases  the  cost  of  maintaining  the  document  through  manual  revision  and 
distribution.  This  cost  includes  the  personnel  time  required  to  process  the  change  request,  for 
engineers  and  writers  to  evaluate  the  change  request,  to  rewrite  the  procedure,  and  to  publish  the 
manual  revision.  From  the  perspective  of  the  operator,  poor  maintenance  documentation  may  be 
associated  with  higher  maintenance  costs.  These  costs  are  a  result  of  slowed  maintenance, 
reduced  revenue  due  to  aircraft  downtime,  resulting  wear  and  damage  to  components,  personal 
injury,  and  the  increased  time  required  to  process  revised  documentation  and  ensure  that  these 
changes  are  distributed  promptly  to  maintenance  technicians. 

When  compared  to  the  potential  costs  (to  aircraft  operators)  associated  with  a  poor  maintenance 
procedure,  the  costs  of  validating  a  procedure  are  relatively  small.  For  example,  it  is  possible  to 
estimate  the  cost  of  reviewing  a  procedure  by  making  several  assumptions.  First,  assume  the 
review  process  takes  4  hours  and  the  cost  of  the  personnel  is  roughly  $200  (average  salary  of  $80 
for  an  engineer  times  the  effort  of  an  estimated  2.5  employees).  The  estimated  total  cost  would 
equal  $800.  Next,  consider  the  cost  savings  to  an  operator  if  that  review  results  in  changes  to  a 
procedure,  reducing  the  total  time  to  perform  the  procedure  by  4  hours.  This  is  time  that  would 
have  been  spent  researching  the  discrepancy  in  the  manual.  Assuming  an  industry  labor  rate  of 
$60,  personnel  cost  of  $150  (average  salary  of  $60  times  the  effort  of  an  estimated  2.5 
employees)  and  4  hours,  the  total  cost  savings  would  equal  $600.  If  an  operator  maintains  25 
aircraft  of  this  type,  the  total  cost  savings  would  be  $1 5,000  for  just  one  procedure.  The  potential 
costs  increase  substantially  when  one  considers  both  the  number  of  procedures  in  a  manual  that 
might  be  improved  to  a  similar  degree  and  the  total  number  of  aircraft  in  service.  Although  the 
costs  associated  with  validating  the  procedure  may  be  high,  in  some  cases  it  should  be  weighed 
against  its  impact  on  safety  and  cost  to  the  industry. 
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3.2  ROLE  OF  THE  FAA  AS  REGULATOR. 


It  is  recommended  that  industry  should  be  strongly  encouraged  to  adopt  an  MSG-3  system  and 
logic  to  identify  what  maintenance  procedures  should  be  validated.  Manuals  are  a  required  part 
of  the  approved  maintenance  process,  but  with  the  exception  of  the  flight  manual  and  the 
structural  repair  manual,  they  do  not  require  certification  of  the  information  they  contain. 
Considering  the  size  of  the  manual  and  the  need  to  provide  timely  information  to  operators, 
certification  of  the  maintenance  manual  is  not  required.  Manufacturers  should  implement  a 
process  for  identifying  critical  maintenance  procedures  and  methods  for  validating  the  relevant 
maintenance  documentation.  This  process  would  include  methods  for  identifying  which 
procedures  should  be  evaluated,  identifying  which  form  of  evaluation  testing  would  be 
appropriate,  establish  standards  for  performing  evaluations,  and  a  system  for  recording  user 
feedback,  and  actions  taken.  This  information  could  be  used  to  justify  the  validation  efforts  of 
each  manufacturer.  Rather  than  a  generic  certification  process  to  which  all  manufactures  would 
be  subject,  this  would  allow  each  manufacturer  to  tailor  the  evaluation  process  to  problems 
unique  to  their  organization  as  well  as  those  of  their  customers.  The  role  of  a  regulator  is  to 
oversee  the  implementation  and  periodically  review  this  process  to  ensure  compliance. 

3,3.  THE  MSG-3  PROCESS:  A  MODEL  FOR  THE  DEVELOPMENT  OF  TECHNICAL 
DOCUMENTATION. 

In  the  same  way  that  reliability  analysis  of  components  requires  tracking  the  performance  of 
those  components  over  time,  an  effective  usability  analysis  requires  tracking  the  history  of  user 
problems,  the  attempts  to  fix  those  problems,  and  the  outcome  of  those  attempts.  The  following 
is  a  review  of  how  this  information  can  be  used  as  part  of  larger  program  to  identify  procedures 
that  should  be  validated. 

An  example  of  a  system  similar  to  that  being  proposed  is  the  Maintenance  Steering  Group 
process  for  Reliability  Centered  Maintenance  (RCM)  [6  and  7].  RCM  analysis  is  used  to 
establish  a  maintenance  and  inspection  program  tailored  to  the  failure  history  of  each  aircraft  part 
or  system.  By  matching  the  inspection  and  maintenance  schedules  to  reliability  data,  both  the 
cost  of  maintenance  and  the  potential  for  maintainer-introduced  damage  and  error  are  reduced. 
Key  components  of  the  RCM  analysis  are  the  identification  of  critical  systems  and  components, 
the  tracking  of  system  component  failures  and  replacements,  and  the  periodic  review  of  program 
effectiveness.  When  properly  implemented,  the  MSG-3  process  provides  maximum  safety  at 
minimal  cost. 

While  the  goal  of  the  MSG-3  logic  is  to  develop  a  maintenance  program  and  schedule,  a  similar 
process  could  be  used  to  guide  a  validation  program  for  maintenance  technical  manuals.  Two 
key  components  of  the  MSG-3  process,  the  decision  logic  used  to  identify  the  potential  impact  of 
failures  and  the  tracking  of  reliability  data,  could  easily  be  adapted  to  guide  the  validation  of 
technical  data.  By  evaluating  the  level  of  risk  posed  by  a  given  procedure  (both  to  safety  and 
economics),  the  need  for  validation  can  be  determined.  Once  the  need  for  validation  is 
identified,  the  risk  potential  can  be  further  delineated  to  determine  the  most  appropriate  method 
of  validating  that  procedure.  In  addition,  by  tracking  the  history  of  procedure  difficulties  and 
validation  efforts,  any  need  to  modify  the  program  can  be  periodically  addressed.  Rather  than 
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being  limited  to  an  initial  requirement  of  developing  a  maintenance  program,  RCM  analysis  uses 
the  continued  accumulation  of  operating  experience  to  improve  the  safety  and  efficiency  of  the 
program. 

The  MSG-3  decision  logic  begins  by  assessing  each  potential  functional  failure  in  terms  of  both 
safety  and  economic  impact.  It  is  worth  noting  that  this  process  begins  with  an 
acknowledgement  of  the  inherent  level  of  safety  and  reliability  of  a  system.  Just  as  all 
mechanical  components  will  eventually  fail,  so  too  will  human  systems  fail.  Once  a  lower  bound 
to  system  reliability  is  realized,  the  objectives  become  the  achievement  of  that  inherent  limit,  the 
restoration  of  that  level  when  deterioration  occurs,  and  the  redesign  of  the  system  when  that  limit 
is  no  longer  acceptable.  All  of  which  should  be  accomplished  at  a  minimum  cost,  both  for  the 
completion  of  the  task  and  for  any  related  failures.  Not  all  procedures  have  the  same  sensitivity 
to  usability  problems,  nor  do  they  have  the  same  potential  impacts  to  safety  when  problems 
occur.  In  order  to  be  efficient  and  cost-effective,  validation  must  be  tailored  to  the  specific  task. 

Just  as  maintenance  significant  items  (MSIs)  and  structurally  significant  items  (SSIs)  are 
identified  in  the  MSG  process,  usability-significant  procedures  should  be  identified  in  the 
technical  documents.  Significance  is  evaluated  based  on  potential  outcomes  resulting  from  a 
usability  problem  with  the  information  used  to  perform  the  procedure.  Usability-significant 
procedures  would  be  those  procedures  where  a  usability  problem  or  error  could  lead  to: 

•  a  safety-critical  problem  (on  the  ground,  in  the  air,  or  threat  to  maintenance  personnel). 

•  a  failure  that  is  not  easily  recognized. 

•  a  significant  operational  impact. 

•  a  significant  economic  impact. 

Once  the  potential  risk  of  a  usability  problem  is  evaluated  and  usability-significant  procedures 
are  identified,  procedures  should  be  evaluated  to  identify: 

•  the  range  of  potential  usability  problems  that  could  occur. 

•  the  effect  of  those  problems. 

•  potential  reasons  for  those  problems. 

The  success  of  the  system  depends  in  large  part  on  the  collection  and  analysis  of  how  and  when 
components  or  aircraft  systems  fail.  By  analyzing  the  details  of  system  failures,  trends  are 
identified  that  allow  operators  to  adjust  maintenance  schedules,  minimizing  the  potential  impact 
of  failures.  By  analyzing  how  and  when  technicians  encounter  usability  problems  with  a  manual, 
it  is  possible  to  adjust  the  manual  evaluation  process  to  focus  on  those  areas  that  would  be  most 
adversely  affected  by  usability  problems. 

Some  procedures  may  be  selected  for  validation  prior  to  aircraft  delivery  based  on  prior 
experience.  Others  would  be  targeted  for  validation  in  the  event  that  operational  data  indicates 
that  it  may  be  the  source  of  usability  problems.  It  follows  that  manual  validation  should  not  be  a 
criteria  for  aircraft  certification.  This  would  allow  for  rarely  performed  procedures  or  those 
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pertaining  to  repairing  damaged  components  to  be  validated  when  first  encountered  in  the 
operational  environment. 

4.  ADDITIONAL  COMMENTS. 

4.1  THE  TRAINING  OF  MAINTENANCE  PERSONNEL  AND  TECHNICAL  WRITERS. 

Although  not  central  to  the  research  objectives  of  this  project,  the  question  of  training  with 
regard  to  technicians  and  technical  writers  was  raised  by  manufacturers  and  technicians. 
Concern  was  expressed  that  newly  hired  technicians  may  receive  little  or  no  training  on  using 
technical  manuals  and  that  this  may  contribute  to  some  of  the  problems  reported  with  searching 
for  information  and  navigating  technical  documents  [5].  This  matter  highlights  the  broader  issue 
regarding  the  skills  of  new  technicians.  Some  concern  has  been  voiced  that  technicians  entering 
the  field  have  fewer  and  poorer  skills  than  in  the  past.  For  example,  the  director  of  maintenance 
at  a  major  domestic  airline  reported  that  only  approximately  half  of  some  1600  potential 
employees  passed  a  basic  mechanics  skills  test  [8].  The  potential  importance  of  this  should  not 
be  underestimated.  The  low  accident  rate  of  American  aviation  is  partly  a  result  of  the 
experience  level  of  maintenance  technicians,  their  quality  of  training,  and  the  quality  of  the 
technical  information  used  to  maintain  the  aircraft.  A  decline  in  any  of  these  factors  may  be 
expected  to  erode  the  safety  margins  we  currently  rely  upon. 

Similarly,  many  of  the  technical  writing  groups  have  reported  difficulties  finding  employees  with 
training  in  technical  writing  who  are  generally  familiar  with  aircraft  and  their  operation.  In 
several  cases,  manufacturers  attempt  to  compensate  for  this  by  hiring  a  mix  of  former 
technicians,  engineers,  and  writers  to  develop  maintenance  documents.  The  aviation  technical 
writing  community  would  benefit  from  standardized  training  to  ensure  a  common  basic  skill 
level.  This  training  could  be  adapted  to  the  needs  of  the  company,  as  well  as  needs  of  the  users 
in  response  to  their  feedback  regarding  manual  reliability. 

4.2  RECENT  DEVELOPMENTS. 

In  response  to  the  findings  and  recommendations  of  this  research  effort,  an  Associates  of  Arts 
(AA)  degree  program  was  established  at  local  Wichita  area  colleges  with  coordination  from 
WSU.  The  AA  program  was  developed  as  a  result  of  a  joint  venture  between  local  aircraft 
manufacturers  (Raytheon,  Lear,  and  Cessna),  educational  institutions  (Wichita  Area  Technical 
College  and  Wichita  State  University),  the  Wichita  Chamber  of  Commerce,  and  a  consulting 
business  (Entrepreneurial  Foundations,  LLC).  This  joint  venture  led  to  the  development  of 
certificate  and  degree  programs  for  aviation  technical  writers.  Through  the  collaborative  efforts 
of  this  group,  it  will  provide  an  environment  for  teaching  the  necessary  skills  of  technical 
writing,  as  well  as  aircraft  systems  operation  and  maintenance  procedures.  Classes  began  in  the 
fall  of  2002. 

5.  SUMMARY. 

A  series  of  recommendations  have  been  proposed  that  are  believed  would  improve  the  quality  of 
maintenance  technical  documentation.  In  most  cases,  the  recommendations  follow  directly  from 
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suggestions  made  by  representatives  from  the  aircraft  industry  that  participated  in  the  project. 
Some  of  the  recommendations  require  minor  modifications  to  existing  programs  or  efforts  within 
the  manufacturers,  while  others,  such  as  the  prioritization  of  maintenance  procedures  for 
validation,  require  the  cooperation  of  manufacturers,  regulators,  and  operators  to  be  realized. 
These  recommendations  are  an  attempt  to  shift  from  a  reactive,  production-centered  approach 
towards  a  more  proactive,  user-centered  process  of  generating  technical  documentation  that 
focuses  resources  on  those  aspects  of  the  manual  that  may  have  the  greatest  impact  on  usability, 
safety,  and  economics. 
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